Unsupervised Learning of the Morphology of a Natural Language

This study reports the results of using minimum description length (MDL) analysis to model unsupervised learning of the morphological segmentation of European languages, using corpora ranging in size from 5,000 words to 500,000 words. We develop a set of heuristics that rapidly develop a probabilistic morphological grammar, and use MDL as our primary tool to determine whether themodiŽcations proposed by the heuristicswill be adopted ornot. The resulting grammar matches well the analysis that would be developed by a human morphologist. In the Žnal section, we discuss the relationship of this style of MDL grammatical analysis to the notion of evaluation metric in early generative grammar.